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Use of Testing/Evaluation Information for School Improvement 
Glynn Ligon, Austin (TX) Public schools 



This perspective of how testing and evaluation information is. 
used and could be used for school improvement is based partly 
upon what we have managed to accomplish within the Austin 
Public Schools over the past decade and a half and partly 
upon what we desire to accomplish e Theoretically, there are 
four conditions that are seen as necessary for effective use 
of evaluative information by school administrators ^ Beyond 
these four conditions, this paper describes two dimensions 
that influence how evaluators approach the reporting of 
information. Finally, a few examples are provided of how we 
report outcomes to schools within the cont3Xt of the factors 
that typically influence achievement. 

Conditions That Encourage the Use of Testing/Evalxiation 
Information for School Improvement . The diagram below shows 
that the first key to utilization is the existence of a real 
need for information. In addition, there must exist a degree 
of trust of the evaluator by the school administration. The 
school administration must understand the information they 
are being provided in order to apply it. Finally, there must 
develop a practice of using the information. 




Needt We have tried several times over the years to 
provide information that was clearly useful to 
schools and found that the information is greatly 
underutilized or ignored. ' Often this is not a 
reflection upon the potential for use of the 
information or even upon the acceptance by the 
school administration of the information . For 
example, the school goal-setting process at times 
has been approached by principals as a nvecessary 
hurdle that can be overcome with a quick and shallow 
statement that can be placed on file and never 
brought out for accountability use. This is an 
example of a requirement that calls for information, 
but with such a low level of need for timely, useful 
information that the principals ' other priorities 
win out. 

Trust . When our Office of Research and Evaluation 
b' tan in 1973, there was not just an absence of 
tr* t, but a real distrust of the evaluators and 
the^r information. This distrust was somewhat 
justified by the lack of experience and track record 
of the evaluators, and by the fact that how 
evaluation and expanded testing information was to 
be used was a mystery. For school improvement to 
result from evaluation information, school 
administration must have a high level of trust not 
only in the information that is provided, but also 
in the providers of that information. 

Understanding . Related closely to trust, is the 
need for the school administration to understand the 
testing/evaluation information with which they are 
working. Beyond the uncomfortable feeling one has 
when working with numbers that are not fully 
understood, there is the real issue of potential 
misuse oZ the information in goal setting or 
accountability. Principals must understand the data 
in order to accept the conclusions of the e valuator 
or to form independent judgements. Principals must 
understand the data in order to set realistic goals, 
targeted at real areas of need, and measurable by 
appropriate statistics. 

Practice. Theories are fine, but practice is what 
makes a difference. In school improvement, getting 
the use of evaluation/testing information into the 
mainstream of decision making is a key. Obviously, 
principals can run schools without 

testing/evaluation information. Successful practice 
of using information within a school^ s planning 
process perpetuates use . 
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The interesting thing about our history in Austin is that the 
above order has not been followed. In fact breakouts of each 
of these conditions have occurred over the years, but a 
systemwide embracing of the use of testing/evaluation 
information for school improvement had to wait for the 
pressures of education reform, tight budgets, and the 
challenge to make the return to neighborhood schools work. 
The conditions of trust and understanding have developed over 
the years — with more improvement greatly needed. The 
condition of need has come and gone, but had never come as 
seriously as in the past year or so. Now we are moving to 
fine tune our practice to achieve a better match between the 
volumes of reports that schools receive and the practical use 
to which they can put the information contained in those 
reports . 

The two dimensions which are important to understand when 
considering the use of testing/evaluation information for 
school improvement are discussed below. 

1. Level of Preparation of the Data. This dimension 
ranges from pure raw data, such as individual 
studenv. test scores , to f ormularized school goals 
that are predetermined by the computer and fed into 
a word processing document for the principal to 
sign . A fascinating internal debate that continues 
is whether a principal must struggle with some 
relatively raw data, summarize it, graph it, 
basically personally squeeze out the trends 
in order to understand the information; or whether 
it: is better to save the nonevaluators the agony, 
and quite frankly to avoid the errors, and to hand 
the principals neatly packaged summary reports that 
have done all the work of organizing the data and 
present a simplistic bottom line that the principal 
can see immediately* The scarcity of principals^ 
time has tended to move us toward the latter , and 
our attempts at the former have emphasized how time 
consuming it is to take nontechnical managers 
through a semitechnical exercise . 

Obviously, the technology is here that can hand a 
principal a final set of school goals , accurately 
Targeted at the school's needs and accurately stated 
in the correct terms — but will the principal accept 
ownership of the goals and express gratitude for the 
savings cf time; or will the principal resist the 
goals , regardless of their appropriateness . This 
relates back to two elements of our conditions for 
utilization. With adequate trust and understanding, 
the principals and the e valuators can negotiate the 
proper position along this dimension from raw data 
to formula goals. 
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2, Ultimate Use> Setting a school goal or improving 
instruction within a school are seldom the ultimate 
uses for testing/evaluation information. This 
second dimension of ultimate use ranges from 
personal/private introspection on the part of the 
principal, past public discussion and 
accountability, to the evaluation of the job status 
of the principal. This is an incredibly important 
dimension to understand. How a system for 
utilization is designed and how the information is 
reported should be determined by this dimension. 
For example , before we moved closer to public 
accountability and evaluation of the principal, 
there was little need for every school to follow 
certain standards for their school goals. Every 
principal could select goal areas, measures of 
success, and reporting formats. Now the 
accountability reality forces us to standardize 
goals, measures, and reports. 



From this second dimension of use, comes the reality that as 
accountability increases, so does the responsibility for the 
accountability measures to be real indicators of a school's 
success. We have all talked for years of the inequities of 
comparing mean test scores for two schools with different 
income levels in their student populations. However, now we 
must become even more sophisticated in treatment of these 
context variables. Our district developed the Report on 
School Effectiveness (ROSE) in response to this admonition. 
Attachment A describes ROSE and our attempt to compare a 
school's actual test scores with a prediction of that 
school's scores based upon key context variables. This trend 
is now evident in both Texas' statewide testing program and 
in the National Assessment of Educational Progress' expansion 
to state-by-state comparisons. Both testing programs are 
working with ways to present average test scores along with 
the context variables that are known to influence those 
scores . 

Context . Attachment B includes three school profiles we 
produce in an attempt to give principals and others an 
overview of the context variables associated with some very 
general outcome variables. This profile purposely commits 
the sin of oversimplification, because its intent is to 
produce some general rankings that can be examined for 
patterns. The saving grace for this profile is that the 
range of variables shown is wide, thus avoiding contributing 
to simplistic judgements about a school's effectiveness based 
upon a single piece of information. 
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As we continue to refine this type of reporting, drafts have 
been drawn of a school profile that focuses on the correlates 
of an "effective school." This profile will give the 
principals of our 16 priority elementary schools with special 
resources for minority students direct measures of those 
correlates. See Attachment C for the latest draft. 

Details . Details details, details, and more details can be 
found in the hundreds of testing and evaluation printouts 
that arc? produced for schools. These range from alphabetic 
and rank order listings of students by test score to 
classroom summaries by skill areas. Our school system 
provides to principals and teachers an incredible array of 
computer output that they can use for planning, grouping, and 
assessing. However, details tend to be used less than the 
summary reports of means, medians, and percentages within 
ranges for school goal setting; and the trend appears to be 
toward even more summarization of information. A caution 
that we must heed locally is to avoid losing sight of the 
complexity of the context within which these test scores and 
evaluation findings emerge. We must continue to report and 
discuss the details not only as an acknowledgement of the 
complexity of the educational arena, but also as a control 
against making narrow-minded decisions about the relative 
success of different instructional programs and even 
different educators. 
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ATTACHMENT A 
Page 1 of 5 



THE ROSE-THE REPORT ON SCHOOL EFFECTIVENESS 



What is ROSE? 

ROSE, the Report on School Effectiveness, provides information about AISD 
schools that is more than just descriptive. It is the result of a series of 
statistical analyses which answer the question, "How do the achievement gains 
of a school's students compare with those of other AISD students of the same 
previous achievement levels and background characteristics?" Regression 
analysis is used to produce predicted achievement levels in reading and mathe 
matics for each student based on the following characteristics: 



0 Previous achievement level, 

0 Sex, 

0 Ethnicity, 

0 Family income (whether or not the student or a 

sibling received a free or reduced-price lunch), 
0 Whether or not the student's school was impacted by 

desegregation, 
0 Whether or not the student was reassigned by the 

desegregation plan, 
0 Whether or not the student was a transfer student, and 
0 The average pupil/teacher ratio for the student's grade 

at his/her school (elementary only). 

The predicted scores are then compared with the students' actual scores. The 
numbers in parentheses give the average difference between the predicted and 
actual scores in grade equivalents. For exemple, a value of +.10 would mean 
that the students at that grade scored one month higher on the average than 
similar students districtwide. The verbal descriptors, "Exceeded Predicted 
Sain," "Achieved Predicted Gain," and "Below Predicted Gain" are assigned 
according to the statistical significance of the results. If the obtained 
average is far enough above or below the expected value of zero so that it 
would have occurred only 51 of the time or less by chance, then the 
"Exceeded" or "Below" is assigned. 



Wwt Is the purpose of ROSE? 

The pv^oose of ROSE is to improve student achievement in reading and mathe- 
matics through the identification of groups of students who are experiencing 
exceptional success or failure. The identification of these students creates 
an opportunity for improvement in the overall program if practices or condi- 
tions associated with the success or failure of these students can be 
identified. 
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If a school has some students who (ire scoring above the predicted levels in 
reading and mathematics, an examination of the practices of their teachers 
may reveal information which will be useful in improving performance for stu- 
dents in other groups or subject areas. Cases where the students are 
scoring below the predicted level also require close attention so that 
practices or conditions which are retarding student growth can be identified 
and altered. 



Soge Cautions i 

In using ROSE, keep the following points in mind: 

a. ROSE has its greatest value when the results do not entirely match 
your informal assessment; i.e., when it is providing you with new 
information. If the results are the complete opposite of your 
experience, however, then the analyses should be viewed with 
caution. 

b. Test results have been considered only for reading and mathematics. 
Exemplary or poor performance in other areas has not been examined. 

c. ROSE attempts to adjust for as many factors outside the school's 
control as possible. When above- or below-average performance is 
found, additional factors outside the school's control may still be 
operating. Knowledge of the situation at the school is impor- 
tant to a full understanding of the report. 

d. ROSE should be used constructively. The emphasis should be on 
initiating and reinforcing good practices and identifying 
problems. Remember, the purpose is to improve the education of our 
students . 

e. Given that ROSE controls for certain background characteristics, 
some schools with high concentration? of low-income, low-achieviag 
students will be found to exceed predicted achievement at some 
grades, even though their average achievement level is low. It is 
a strength of ROSE that it recognizes the effectiveness of the 
teachers of these 5tuden^:s; however, nothing in the ROSE report 
should be taken 5s an indication that the District is satisfied with 
the achievement of our low-achieving students. Indeed, it is a 
priority goal of the District that low student achievement be 
improved at all grade levels. We expect over time that the effect 
of certain factors now explaining low achievement will have less 
effect on predicted achievement. ROSE may contribute to the 
success of that goal by reinforcing the efforts of effective 
teachers and by highlighting effective practices for others to 
follow. 



ERJC 9^ 



87.30 



ATTACHMENT A 
Page 3 of 5 



f. The statistical significance of the results is influenced by the 
number of students tested; i.e., any given value is more likely to 
represent a real difference from the expected value if it is 
obtained from 100 students rather than 50." Therefore, in some 
cases results that are significant may appear to be less extreme 
than other results that are nonsignificant if the sizes of the 
groups differ greatly. 



School Characteristics Inforaation 

The values for the school characteristics listed on the ROSE may differ from 
those listed in your school achievement profiles or elsewhere. The ROSE 
values are based on the population used in doing the analyses and therefore 
may not exactly reflect the total school population. 
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REPORT ON SCHOOL EFFECTIVENESS IROSHl FOR 1986-67 
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REPORT ON SCilOOL EFFECTIVENESS (ROSE), 1982-1987 
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AUSTIN INDEPENDENT SCHOOL DISTRICT 



EFFECTIVE SCHOOL STflNDflRDS 



To be identified as an effective school, a Priority School must 
meet each of the following standards for two consecutive years. 

§*!i^gD* SfefegQdancg 

An effective school is one with an average daily attendance <flDA> 
of 95:^. or more. 



Staff StteiQdanca 

Each teacher at an effective school has an absence rate of 5 or 
fewer days of sick and personal leave each year. Teachers who 
take maternity leave or have catastrophic illnesses may be 
ip'xcluded. 



On the TEAMS, effrictiv® schools have 85:4 or more of their students 
mastering all tests. 

For the purpose of ovSiluating this standard, scores will be 
combined by test area across grades. To meet the standard, B5% of 
the students taking each test (mathematics, reading, and writing) 
for a valid score must meet mastery. Therefore, if 83% or more of 
the students reached mastery in reading and mathematics, but only 
83% met mastery in writing, the school would not be classified as 
effective. In addition, any school having £0 or more students 
taking the Spanish TEAMS will be required to reach the 85% mastery 
level on each Spanish test. While this T5AMS standard does not 
address performance comparisons by sex, ethnicity, or income, 
comparisons will be available from a new report being sent by TEA 
to all schools. 



For grades 1-S, the median schoolwide ITBS Composite score should 
be at least the SOth percentile, and there should be fewer than 
10% of the students in the bottom quart ile. The data must be 
disaggregated by sex, ethnicity, and income. An effective school 
is equally effective for all groups. For groups with 20 or more 
students, there should fee no more than a 7 percentile point 
difference between groups~males and females, etc. 
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The rules «ippXied to the selection of students for the achievement 
profiles will be used to ©elect students for evaluating this 
standard also* 



Based on a parent questionnaire, seventy-five percet^c or more of 
the parents think an ©ffee^^t-ive school is effect ive* 
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AUSTIN INDEPENDENT SCHOOL DISTRICT 



STflNDflRD FOR IMPROVING SCHOOLS 



Th® effective school standards are long-rang© objectives for the 
Priority -Schools. Until a school meets the standards for an 
effective school, it may be designated as an improving school if 
it meets the standard below. 

fln improving school is one for which the percent^ige of students 
mastering each TEAMS test area (mathematics, reading, and writing) 
meets or exceeds the percentages listed below: 



TEAMS 

_YgaS eiB£QRS2aNQi-.siQ!v|D9RD_ 

'88 7094 Mastery 

»89 7S% Mastery 

'90 80^ Mastery 

»91 as% Mastery 



The percentage is to be calculated in the same way as for the 
effective school TEAMS standard. Also, schools with SO or more 
students tested in Spanish must meet the standard in each 
language. 
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AUSTIN INDEPENDENT SCHOOL DISTRICT 



PERIODIC INFORMATION FOR MONITORING PROGRESS 



Two o-f the five standards can be monitored on an ongoing basis 
during the school year. The following describes the committee- 3 
recommendations for information for ongoing monitor ing« 

§tadenfe StfeSGdSDSS 

Each six weeks* Printout will provide the average percentage of 
the days enrolled that students were in attendance. Averages will 
be provided by teacher and by sex and ethnicity for each grade? .nr.-ri 
the school as a whole* 

Staff 81i22^§DSS 

At the end of October, January, March, and for the entire year. 
The number of hours of sick and personal leave <and total) will be 
reported by employee. The average number of hours of each type of 
leave avid ths tctsl for teacher^ will also bm pre^^nted for the 
school and the District. 



